core: analyze every dialect through the new analysis core - #4538
Open
kyleconroy wants to merge 7 commits into
Open
core: analyze every dialect through the new analysis core#4538kyleconroy wants to merge 7 commits into
kyleconroy wants to merge 7 commits into
Conversation
`sqlc analyze` ran ClickHouse and GoogleSQL on the core catalog and analyzer while PostgreSQL, MySQL and SQLite still went through the legacy compiler. This routes all five dialects through the core. - compiler: NewCompiler takes options; WithCoreAnalysis, which the analyze command passes, builds a core catalog seeded with the engine's dialect and skips the legacy catalog and the analyzer connection entirely. ClickHouse and GoogleSQL keep doing so unconditionally. `generate`, `vet` and `compile` are untouched. - core/seed: a declarative description of a dialect — its types and their aliases, the operators and casts between them, and the functions it ships with — that Apply turns into catalog rows. PostgreSQL, MySQL and SQLite seeds are built on it, reusing each engine's existing function catalog (pg_catalog and the MySQL and SQLite stdlibs). - core: literals take the type each dialect names rather than PostgreSQL's, a type a schema declares gains the dialect's comparison operators, and array types are named after their element. - core/schema: enums, functions, views, CREATE TABLE AS, ALTER TABLE and the rename statements now load into the catalog, and a column's array-ness is read from either the type name or the column. - core/analyzer: CTEs, set operations, subqueries in FROM and as expressions, correlated references, functions in FROM, IN, BETWEEN, CASE, COALESCE, array expressions, casts and output-name references in GROUP BY and HAVING. Overloads are chosen by argument type, polymorphic return types resolve to the argument's, and an unknown function or operator leaves the expression untyped instead of failing the query. The SQLite golden changes because the core reports the type name the catalog stores, which is lower case. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
Each engine now embeds a dialect.json and hands it to seed.Dialect, which parses it into a Spec and applies it. An unknown field is an error, so a misspelled key is caught where it is written rather than silently ignored. ClickHouse and GoogleSQL move onto the seed package at the same time, replacing their hand-rolled type and operator loops. Both gain the two things the shared spec gives every dialect and their loops left out: the type each literal takes on, and a count() aggregate. Generated function catalogs stay in Go — pg_catalog and the MySQL and SQLite stdlibs are passed to seed.Dialect rather than restated as JSON — and a dialect can declare a handful of functions inline where there is no generated list. A new test loads every engine's dialect.json into a catalog and checks that its literals and a sample of its type spellings resolve and compare. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
State the testing strategy in CLAUDE.md — a change is exercised by running sqlc the way a user does, against a schema, a query file and a committed golden, rather than by a unit test on an internal function — and describe how an endtoend case is laid out. Drop the seed package's unit test accordingly. The dialect descriptions it covered are exercised by the analyze cases under internal/endtoend/testdata, which run every dialect through the catalog it seeds. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
…to it A dialect is now a directory of JSONL files rather than one JSON document: dialect.jsonl names it and carries its rules, and types, operators, casts and functions each get a file of one record per line. Records are applied as they are read, so a function list running to thousands of entries is never held in memory as a whole, and a dialect only writes the files it needs. With a record-per-line format the engines' standard libraries fit in it, so they move out of Go: - MySQL and SQLite: stdlib.go held the function list as Go literals. It is now functions.jsonl, and defaultSchema reads it. 6.6k lines of Go gone. - PostgreSQL: pg_catalog.go carried ~2700 functions ahead of the system tables. Those move to the dialect directory too, leaving the generated file to the tables alone — 30k lines of Go gone — and sqlc-pg-gen learns to write the JSONL, so a regeneration produces the same layout. Each list is read once per process and feeds both the analysis core and the catalog the legacy compiler builds, so there is one copy of the data rather than one per consumer. The binary drops about 3 MB. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
The file holds one record, so a line-oriented format bought nothing and read oddly next to the lists it sits with. It is now an ordinary JSON document, indented and diffable. The lists stay JSONL, which is where streaming a record at a time pays. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
pg_catalog's 139 tables and information_schema's 69 were the last of PostgreSQL's catalog held as generated Go, and the analysis core had no notion of a relation it did not read from the user's schema — so a query against information_schema.columns failed to analyze. They are now records in a dialect's relations.jsonl, keyed by schema: sql_class is what the core stores them in, and "relations" is the friendly name for it the way "functions" is for sql_proc. sqlc-pg-gen writes the file, the core seeds classes and attributes from it, and the two schemas the legacy compiler builds are read back from the same records — which takes pg_catalog.go and information_schema.go out of the tree entirely. Relations are seeded on demand rather than up front. A dialect's system catalogs are thousands of columns that most queries never name, and loading them for every invocation cost 40ms; the catalog now loads them the first time a query names a schema it does not have. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
Relations are seeded with the rest of the dialect again, so every record in a dialect directory is applied as it is read and the catalog has no notion of a seed that runs later. The cost this was avoiding — about 40ms per invocation to install PostgreSQL's system catalogs — comes back, and will be dealt with another way. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011MnoUabwBWW9gaEn2Nj7eG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sqlc analyzeran ClickHouse and GoogleSQL on the core catalog and analyzer while PostgreSQL, MySQL and SQLite still went through the legacy compiler. This routes all five dialects through the core, and — since a dialect now has to be described rather than coded — moves each engine's type system and standard library out of Go and into data files the core and the legacy compiler both read.generate,vetandcompileare untouched: they still use each engine's own analysis path.Analyzing every dialect through the core
NewCompilertakes options;WithCoreAnalysis, which the analyze command passes, builds a core catalog seeded with the engine's dialect and skips the legacy catalog and the analyzer connection entirely. ClickHouse and GoogleSQL keep doing so unconditionally.int4/numeric/text/bool, which only PostgreSQL has); a type a schema declares — an enum, or one of SQLite's free-form type names — gains the dialect's comparison operators; array types are named after their element.CREATE TABLE AS,ALTER TABLEand the rename statements now load into the catalog, and a column's array-ness is read from either the type name or the column. Five new catalog queries back theALTER TABLEmutations, generated by sqlc itself.FROMand as expressions (including correlated ones), functions inFROM,IN,BETWEEN,CASE,COALESCE, array expressions, casts, and output-name references inGROUP BY/HAVING. Overloads are chosen by argument type, polymorphic return types (max(anyelement)) resolve to the argument's type, and an unknown function or operator leaves the expression untyped instead of failing the whole query.Dialects as data
A dialect is a directory each engine embeds:
The lists are JSONL and are applied a record at a time as they are read, so a function list running to thousands of entries is never held in memory as a whole;
dialect.jsonholds one object, so it is read whole. Unknown fields are rejected, which catches a misspelled key where it is written. Any list may be left out — ClickHouse ships three files, PostgreSQL six.With a record-per-line format the engines' standard libraries fit in it, so they moved out of Go:
dolphin/stdlib.godolphin/dialect/functions.jsonlsqlite/stdlib.gosqlite/dialect/functions.jsonlpostgresql/pg_catalog.gopostgresql/dialect/{functions,relations}.jsonlpostgresql/information_schema.gopostgresql/dialect/relations.jsonlEach list is read once per process and feeds both the analysis core and the catalog the legacy compiler builds, so there is one copy of the data rather than one per consumer.
sqlc-pg-genwrites the two PostgreSQL files, so a regeneration reproduces this layout rather than reverting to Go. The binary drops about 3 MB.Moving the system catalogs across also closed a gap: the core had no notion of a relation it did not read from the user's schema, so a query against
information_schema.columnsorpg_catalog.pg_classcould not be analyzed at all. It can now.Behavior change
analyzereports type names as the catalog stores them, in lower case — so SQLite'sINTEGERis nowinteger. Theanalyze_basic/sqlitegolden reflects that, anddocs/howto/analyze.mdcalls it out.Testing
CLAUDE.mdnow states the testing strategy this follows: new work is covered by end-to-end cases underinternal/endtoend/testdata, not by unit tests, and the file describes how a case is laid out. Sixteen analyze cases are added or extended across the five dialects, covering joins, aggregates, CTEs,RETURNING, multi-tableDELETE,sqlc.arg/narg/slice, and the PostgreSQL system catalogs.As a wider check I ran
sqlc analyzeover every PostgreSQL/MySQL/SQLite schema+query pair inexamples/andinternal/endtoend/testdata— 262 pairs, of which 254 pass, up from 178 when the three dialects were first pointed at the core. Roughly twelve of the remaining are intentionally-invalid fixtures (invalid_table_alias,sqlc_arg_invalid,relation_does_not_exist, MySQL files written with$1). The genuine gaps left are recursive CTEs, MySQL schema-qualified column references, SQLite virtual tables and table-valued functions (json_each), unqualified references to system tables that rely onsearch_path, and one MySQL self-join that references a table by a name it aliased away.TestReplay,TestParse,TestFormatandTestJsonSchemapass — thegenerategoldens are what prove the standard-library conversion kept the data intact, since function return types drive codegen. TheTestValidSchemafailures in my environment are pre-existing: no databases run there, and the count is identical onmain.Left for later
contrib/extension catalogs are still generated Go; converting them needs a live PostgreSQL with each extension installed.